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Abstract 


Section  1071  of  the  National  Defense  Authorization  Act  for  Fiscal  Year  2013  required  the 
Director  of  the  White  House  Office  of  Science  and  Technology  Policy  (OSTP),  working  with 
the  Secretary  of  Defense  and  the  Administrator  of  the  National  Aeronautics  and  Space 
Administration  (NASA),  to  conduct  a  study  on  the  ability  of  the  national  test  and  evaluation 
(T&E)  infrastructure  to  effectively  and  efficiently  mature  hypersonic  technologies  for  defense- 
related  systems  development  in  the  short  and  long  term.  It  further  required  the  Secretary  of 
Defense  to  submit  a  report  to  the  appropriate  congressional  committees  containing  the  results 
of  this  study,  together  with  a  plan  for  requirements  and  proposed  investments  to  meet 
Department  of  Defense  (DoD)  needs  through  2030.  The  Institute  for  Defense  Analyses  (IDA) 
supported  both  the  OSTP  and  DoD  efforts,  and  was  subsequently  tasked  to  quantify  the 
projected  savings  that  could  accrue  to  each  of  three  conceptual  conventional  hypersonic 
system  development  programs  if  the  proposed  T&E  capability  enhancements  presented  in  the 
DoD  plan  submitted  to  Congress  were  available  at  the  beginning  of  Milestone  (MS)  A.  As  part 
of  its  analysis,  IDA  utilized  an  IDA-developed  approach  for  valuing  the  closure  of  critical  T&E 
capability  gaps  based  on  the  potential  programmatic  cost  and  schedule  savings  that  could 
reasonably  be  expected  to  accrue  from  the  start  of  MS  A  through  the  end  of  MS  C  by  funding 
the  five-year,  $350  million  T&E  infrastructure  investment  proposed  in  the  DoD  plan.  This 
expansion  of  the  cost-benefit  “control  volume”  to  include  projected  system  development 
savings  proved  successful  in  justifying  and  securing  full  funding  for  the  proposed  DoD  plan. 
This  paper  summarizes  the  IDA  approach  and  its  application  to  making  the  business  case  for 
closing  capability  gaps  in  other  aerospace  research,  development,  test,  and  evaluation  areas. 


I.  Introduction 


A.  Problem  Statement 

State-of-the-art  ground  and  flight  test  facilities,  computational  techniques,  testing  tools  and  technologies,  and  a 
skilled  workforce  are  the  “tools  of  the  trade”  for  the  aerospace  research,  development,  test,  and  evaluation  (RDT&E) 
community — much  like  hammers,  nails,  screwdrivers,  and  saws  are  for  the  building  and  construction  industry.  But 
despite  the  unarguable  fact  that  the  development  of  new  aerospace  products  requires  a  robust  and  continuing 
commitment  to  foundational  research  and  development  (R&D) — which,  in  turn,  requires  having  available  the 
computational  and  experimental  capabilities  essential  for  managing  risk  during  both  initial  system  development  and 
subsequent  upgrades — justifying  investments  to  enhance  the  test  and  evaluation  (T&E)  capabilities  needed  to 
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effectively  and  efficiently  field  future  military  and  commercial  systems  has  historically  proven  to  be  a  daunting  task 
for  facility  owners  and  operators.  If  the  requirement  is  so  clear,  why  is  the  justification  so  daunting? 

One  reason  is  that  the  management  of  United  States  (U.S.)  test  capabilities — whether  owned  by  government, 
industry,  or  academia — is  largely  stove-piped  and  subject  to  organizational  budget  pressures  and  constraints.  This  is 
further  compounded  by  the  fact  that  test  facilities  and  test  capabilities  have  historically  been  treated  as  cost  centers, 
with  their  current  and  future  existence  determined  by  a  return-on-investment  (ROI)  largely  predicated  on  capacity 
utilization  metrics  (such  as  occupancy  rate  or  billable  test  hours  per  year)  or  anticipated  demand  (projected  usage  rates 
associated  with  existing  or  new  systems).  That  the  test  community  has  struggled  for  decades  to  garner  support  for 
increased  RDT&E  infrastructure  investments  is  proof  these  metrics  do  not  generate  a  compelling  business  case. 

In  the  annual  head-to-head  competition  for  funds  between  R&D  programs  and  T&E  infrastructure  enhancements, 
R&D  programs  almost  always  win  the  battle.  The  large  number  of  T&E  facilities  that  have  either  been  mothballed  or 
closed  over  the  past  three  decades  because  their  business  cases  failed  provides  testimony  to  this  reality.  If  the 
justification  could  be  successfully  made  for  T&E  infrastructure  investments  based  on  utilization  metrics  alone,  the 
aerospace  RDT&E  infrastructure  would  be  owned  and  operated  by  industry  as  a  profit  center.  Absent  a  new  and  more 
compelling  methodology  for  justifying  RDT&E  infrastructure  investments,  the  United  States  is  at  risk  of  not  having 
the  required  computational  and  experimentation  capabilities  it  needs  for  future  aerospace  product  development. 

B.  Historical  Perspective 

Because  test  facilities  are  often  unique,  as  well  as  expensive  to  construct  and  sustain,  the  U.S.  government  has 
traditionally  owned  and  operated  the  nation’s  RDT&E  infrastructure  (wind  tunnels,  test  cells,  laboratories,  ranges, 
etc.) — with  a  majority  of  the  larger  Department  of  Defense  (DoD)  and  National  Aeronautics  and  Space  Administration 
(NASA)  facilities  having  been  built  during  the  period  from  the  1940s  through  the  1960s.  The  years  from  post-World 
War  II  through  the  late  1970s  were  the  boom  years  for  experimental  testing,  with  the  focus  being  largely  on  applied 
research,  product  development,  and  system  acquisition.  As  the  number  of  system  development  programs  continued  to 
decline  during  the  subsequent  three  decades,  the  focus  on  RDT&E  infrastructure  shifted  to  reducing  expenditures — a 
dynamic  that  was  largely  fueled  by  near-term  budgetary  constraints  and  funding  instabilities,  as  well  as  by  the  belief 
that  emerging  computational  fluid  dynamics  (CED)  capabilities  would  obviate  the  need  for  “brick  and  mortar”  test 
facilities  in  the  not-too-distant  future.  As  a  result,  the  government  focused  its  scarce  resources  on  preserving  an 
outdated  test  infrastructure  rather  than  investing  in  an  updated  infrastructure  capable  of  supporting  future  U.S. 
aerospace  needs. 

The  test  community  tried  to  advise  senior  leadership  that  rapidly  increasing  system  complexity,  declining 
workforce  “hands-on  experience”  (resulting  from  both  attrition  and  the  reduced  number  of  new  system  developments), 
and  the  unrealized  promise  that  CED  would  obviate  the  need  for  test  facilities  were  creating  a  “perfect  storm”  for  the 
national  RDT&E  infrastructure,  but  that  advice  went  largely  unheeded,  particularly  in  the  high-speed  flight  regime. 
Within  the  last  decade,  however,  the  T&E  community  began  sounding  an  alarm  more  loudly  to  senior  decision-makers 
in  DoD  and  NASA  that  infrastructure  investment  shortfalls  of  the  prior  three  decades  were  contributing  to  unintended 
system  development  cost  and  schedule  consequences,  as  measured  by  the  increased  number  of  undetected  defects 
being  discovered  during  flight  testing,  Nunn-McCurdy  breaches,"^  and  program  schedule  slippages.  An  awareness 
subsequently  began  to  emerge  that  facilities  and  test  capabilities  being  reduced  to  minimum  (or  below-minimum) 
levels,  elimination  of  redundancies  (test  capacity),  and  the  willful  elimination  of  often-infrequently-used  (but  unique) 
government  test  capabilities — including  their  associated  T&E  workforces — were  pre-allowing  these  unintended 
consequences  to  occur.  Thus,  the  realization  began  to  grow  within  senior  leadership  that  something  needed  to  be  done, 
and  soon,  to  improve  risk  management  across  the  developmental  T&E  (DT&E)  process  used  to  support  aerospace 
product  development  to  reverse  these  detrimental  trends  in  system  development  cost  and  schedule.  This  was  especially 
true  in  the  hypersonic  flight  regime,  where  facility  demand  has  historically  been  cyclic  and  unpredictable. 

But  the  daunting  challenge  facing  the  T&E  community  regarding  the  successful  justification  of  increased 
investments  mentioned  earlier  remained — how  does  one  win  the  head-to-head  competition  for  funding  between  R&D 
programs  and  T&E  infrastructure  enhancements?  In  2013,  Congress  inserted  itself  into  the  process. 


A  significant  breach  is  when  the  Program  Acquisition  Unit  Cost  (the  total  cost  of  development,  procurement,  and 
construction  divided  by  the  number  of  units  procured)  or  the  Procurement  Unit  Cost  (the  total  procurement  cost 
divided  by  the  number  of  units  to  be  procured)  increases  15  percent  or  more  over  the  current  baseline  estimate  or  30 
percent  or  more  over  the  original  baseline  estimate.  A  critical  breach  occurs  when  the  cost  increases  25  percent  or 
more  over  the  current  baseline  estimate  or  50  percent  or  more  over  the  original  baseline  estimate. 
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C.  A  New  Approach  for  Justifying  Infrastructure  Investments  -  Expand  the  Cost  Savings  “Control  Volume” 

The  Fiscal  Year  2013  National  Defense  Authorization  Act  (hereafter  referred  to  as  The  Act)  [1]  required  the 
Director  of  the  White  House  Office  of  Science  and  Technology  Policy  (OSTP),  working  with  the  Secretary  of  Defense 
(SECDEF)  and  the  NASA  Administrator,  to  conduct  a  study  on  the  ability  of  the  national  Tc&E  infrastructure  to 
effectively  and  efficiently  mature  hypersonic  technologies  for  defense  systems  development  in  the  short  and  long 
term.  For  purposes  of  this  study,  the  national  hypersonic  T&E  infrastructure  was  defined  as  the  ground  test  facilities 
and  open-air  ranges  owned  and  operated  by  DoD  and  NASA,  as  well  as  those  relevant  ground  test  facilities  owned 
and  operated  by  the  private  sector  (industry  and  academia).^  The  Act  further  required  the  SECDEF  to  submit  a  report 
to  the  appropriate  congressional  committees  containing  the  results  of  the  OSTP  study,  an  assessment  of  current  facility 
condition  and  adequacy,  and  an  assessment  of  non-DoD  test  capabilities  that  could  be  leveraged  to  support  defense- 
related  system  development — along  with  a  plan  that  provides  the  test  capabilities  and  investments  needed  to  satisfy 
DoD  hypersonic  Tc&E  requirements  through  2030.  Based  on  the  OSTP-sponsored  study  findings,  conclusions,  and 
recommendations,  DoD  determined  that  an  additional  $350  million  investment  would  be  needed  to  close  the  identified 
test  capability  gaps.  [2] 

In  addition  to  supporting  both  the  OSTP  study  and  DoD  report  efforts  directed  by  Congress,  in  2015,  the  Institute 
for  Defense  Analyses  (IDA)  was  tasked  by  the  DoD  Test  Resource  Management  Center  (TRMC)  to  quantify  the 
projected  savings  that  could  accrue  to  each  of  three  conceptual  hypersonic  system  development  programs  if  the  T&E 
capability  enhancements  proposed  in  the  DoD  plan  were  available  at  the  beginning  of  Milestone  (MS)  A.  As  part  of 
its  analysis,  IDA  utilized  an  IDA-developed  approach  for  valuing  the  closure  of  the  critical  T&E  capability  gaps,  not 
based  on  traditional  capacity  utilization  metrics,  but  on  the  potential  programmatic  cost  savings  that  could  reasonably 
be  expected  from  the  start  of  MS  A  through  the  end  of  MS  C  if  the  five-year,  $350  million  T&E  infrastructure 
investment  proposed  in  the  DoD  plan  were  funded.  The  rationale  behind  the  use  of  an  investment  justification  process 
that  treats  test  capabilities  as  part  of  the  larger  system  development  programs  they  support  is  that  the  cost  of 
infrastructure  sustainment,  improvement,  and  modernization  is  far  less  than  the  cost  of  the  negative  consequences  that 
result  from  going  forward  with  higher  development  risk  created  by  not  having  the  requisite  test  tools  and  capabilities 
to  better  understand  the  underlying  physics  associated  with  the  concept(s)  under  development. 

This  expansion  of  the  cost-benefit  “control  volume”  to  include  projected  system  development  savings,  as  described 
in  Ref.  [3],  proved  successful  in  justifying  and  securing  full  funding  for  the  proposed  DoD  plan.  This  paper 
summarizes  the  approach  used  and  notes  its  potential  application  to  closing  identified  capability  gaps  in  other  military 
and  commercial  aerospace  RDT&E  arenas  across  the  subsonic,  transonic,  supersonic,  and  hypersonic  flight  regimes. 

II.  A  Value-Based  Justification  Process  for  Increased  RDT&E  Capability  Investments 
A.  Justification  Process  Overview 

The  IDA  value-based  justification  process  involves  expanding  the  analysis  “control  volume”  for  valuing  capability 
investments  (as  depicted  in  Eig.  1)  and  then  developing  a  cost  model  for  the  MS  A  through  MS  C  phases  of  a  system 
development  program  based  on  actual  cost,  schedule,  and  flight  test  frequency  data  compiled  from  previously 
successful  major  defense  acquisition  programs  (MDAPs)  of  similar  operational  capability  and/or  system  complexity. 
This  model  is  then  tailored  to  each  conceptual  system-specific  model  to  assess  the  resulting  cost  and  schedule  growth 
for  future  conceptual  development  programs  if  the  proposed  T&E  capability  enhancements  are  not  made. 

Before  describing  the  justification  process  in  more  detail,  potential  users  are  advised  that  this  is  an  extremely 
information-,  time-,  and  analysis-intensive  effort  requiring  substantial  involvement  and  preparatory  work  by  a  variety 
of  study  team  members  with  experience  in  both  strategic  planning  and  large-scale  program  development;  test 
capability  owners;  investment  stakeholders;  and  subject  matter  experts  (SMEs)  having  specific  detailed  knowledge 
and  expertise  in  technology  development  and  demonstration,  DT&E,  and  the  allocation  of  costs  across  the  various 
elements  of  system  development.  Thus,  because  of  the  magnitude  of  the  effort  required,  application  of  this  process 
has  its  greatest  value  and  benefit  when  used  to  justify  larger-scale  ($100h-  million)  investment  augmentations  having 
substantial  national  military  and/or  economic  impacts. 


^  In  this  study,  the  T&E  infrastructure  was  defined  as  including  not  only  the  ground  and  flight  test  facilities  themselves, 
but  also  the  testing  tools,  test  technologies,  computational  capabilities,  and  supporting  T&E  workforce. 


3 


It  is  also  worth  noting  that  this  process  has  the  greatest  potential  for  success  when  the  investment  stakeholder 
funding  the  proposed  enhancements  owns  both  the  RDT&E  infrastructure  being  enhanced  and  the  system  development 
programs  ultimately  utilizing  it — since  the  requisite  expenditures  and  accrued  savings  typically  fall  into  different 
organizational  budget  accounts  (as  in  DoD).  However,  this  process  might  also  be  used  effectively  for  internal 
investment  justifications  by  industry  RDT&E  infrastructure  owners  who  develop  military  and/or  commercial  systems 
as  well  as  for  investment  justifications  in  research  facilities  owned  by  academia,  provided  an  analysis  “control  volume” 
can  be  constructed  that  affords  potentially  significant  financial  benefit  to  the  academic  institution  (such  as  increased 
revenues  derived  from  expanded  student  enrollment  or  successful  competition  for  larger  research  grants). 

B.  Preparing  the  Analysis  Pathway 

Since  the  potential  accrued  savings  are  derived  from  reduced  system  development  time  and  cost,  the  aerospace 
RDT&E  community  needs  to  first  define  what  it  takes  to  develop  the  conceptual  systems  affected  by  the  capability 
enhancements  being  proposed.  In  many  ways,  this  is  a  more  difficult  and  time-consuming  effort  than  performing  the 
business  case  analysis  (BCA)  itself  Preparing  the  pathway  for  the  BCA  involves  three  critical  steps.  The  first  step  is 
getting  the  attention  of  the  decision-makers  by  making  a  compelling  argument  as  to  why  the  proposed  capability  being 
advanced  is  important  to  our  nation,  because  if  the  decision-makers  do  not  agree  that  there  is  a  compelling  argument, 
the  effort  required  to  develop  the  value-based  justification  will  be  a  waste  of  time  and  energy.  The  second  step  is  to 
define  the  proposed  vision,  goals,  and  key  capability  needs  for  the  capabilities  being  advanced.  The  third  step  is  to 
identify  the  capability  gaps  and  evaluate  the  impacts  of  not  closing  those  gaps  on  the  system  development  process. 
The  activities  associated  with  each  of  these  three  critical  steps  are  discussed  in  greater  detail  below. 

1.  Step  1  -  Get  the  Attention  of  the  Decision-Makers 

The  first  step  in  preparing  the  analysis  pathway  is  to  clearly  articulate  why  the  capabilities  being  proposed  for 
enhancement  are  critically  important  to  the  nation  and/or  the  aerospace  RDT&E  community.  Benefits  (or  negative 
consequences)  are  typically  characterized  in  terms  of  their  military  and/or  economic  impacts.  Military  impacts  are 
typically  framed  in  terms  of  how  the  proposed  activity  can  (1)  quantitatively  improve  U.S.  offensive  or  defensive 
operational  capabilities  in  a  timely  manner;  (2)  qualitatively  alter  the  dynamics  of  regional  conflicts;  and/or  (3)  directly 
respond  to  national  security  policies,  strategies,  and/or  guidance.  Economic  impacts  are  typically  framed  in  terms  of 
how  the  capabilities  being  advanced  can  (1)  quantitatively  improve  commercial  aircraft  capabilities — possibly 
opening  new  routes  and/or  global  markets;  (2)  reduce  product  development  time/time-to-market;  (3)  increase  global 
market  share  in  specific  arenas  (e.g.,  single-aisle,  narrow-body  aircraft);  (4)  reduce  individual  vehicle  operational  cost 
or  overall  fleet  life  cycle  cost;  and/or  (5)  (from  a  national  perspective)  improve  the  balance  of  trade  for  the  U.S. 
aerospace  sector. 

It  is  important  to  emphasize  that,  although  articulating  the  military  and/or  economic  impacts  is  a  necessary  first 
step  in  making  a  compelling  argument  to  the  decision-makers,  experience  has  shown  it  is  seldom  sufficient  (on  its 
own  merits)  to  justify  increased  investment.  Nonetheless,  explaining  why  the  capabilities  being  advanced  are  critical, 
and  providing  the  data  to  support  that  position,  are  essential  to  getting  senior  leadership’s  attention,  because  when  it 
comes  to  winning  the  battle  for  funding,  you  don ’t  have  to  outrun  the  bear,  you  only  have  to  outrun  the  person  next 
to  you  (i.e.,  the  other  competing  efforts  being  considered  by  the  decision-makers  for  additional  investment)! 
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2.  Step  2  -  Define  the  Proposed  Vision,  Goals,  and  Key  Capability  Needs 

Defining  the  desired  end-state  (and  what  is  needed  to  achieve  it)  is  the  next  step  in  the  process  of  preparing  the 
analysis  pathway.  Many  times,  this  is  more  difficult  and  time-consuming  than  initially  anticipated  because  it  involves 
more  than  just  identifying  a  vision,  the  target  goals,  and  the  capabilities  needed  to  get  there.  It  also  involves  gaining 
consensus  on  the  desired  end-state  among  the  various  participants  and  stakeholders — often  referred  to  as  socializing 
the  proposed  vision,  goals,  and  key  capabilities. 

The  visioning  process  may  include  developing  sub-visions  for  the  primary  constituent  parts  of  the  overall  vision, 
depending  on  the  nature  and  extent  of  the  capabilities  being  advanced.  For  example,  if  the  vision  is  at  the  enterprise 
level  in  a  specific  flight  regime,  there  could  be  sub-visions  associated  with  ground  testing,  flight  testing,  test  execution, 
and  workforce  development  in  that  flight  regime,  which  collectively  contribute  in  substantive  ways  to  the  achievement 
of  the  overall  vision.  However,  if  the  vision  is  associated  with  a  new  capability  that  cross-cuts  multiple  flight  regimes 
(such  as  the  integration  of  computational  and  experimentation  capabilities),  there  could  be  an  overall  capability  vision 
as  well  as  sub-visions  for  computation,  experimentation,  and  integration  in  each  of  the  individual  flight  regimes. 
Socializing  the  vision  is  vital  because  “if  you  don ’t  know  where  you  are  going,  any  road  will  get  you  there.  ”  [4] 

In  developing  the  target  goals,  two  questions  should  be  answered  in  quantitative  terms:  (1)  What  are  you  trying  to 
accomplish?  and  (2)  By  when?  It  is  safe  to  say  that  most  organizations  involved  in  aerospace  RDT&E — whether 
within  government,  industry,  or  academia — have  planning  documents  that  contain  a  set  of  quantitative  goals  they  are 
pursuing  and  a  projected  time  for  their  achievement.  It  is  also  safe  to  say  that  many  organizations  fail  to  achieve  their 
stated  RDT(&E  goals  on  schedule  (if  at  all)  for  one  of  three  reasons:  (1)  their  target  goals  are  unreasonable  or 
unachievable,  (2)  their  plan  is  inadequately  resourced  to  meet  their  projected  timelines  (thus  making  them  late-to- 
need),  or  (3)  changes  in  organizational  priorities  or  focus  render  the  plan  no  longer  relevant  to  future  organizational 
strategies  or  directions.  In  the  value-based  justification  process,  knowing  the  “by  when”  and  delivering  the  promised 
outcomes  on  schedule  is  vitally  important,  because  capability  enhancements  delivered  late-to-need  to  a  system 
development  program  (i.e.,  after  the  start  of  MS  A)  cannot  use  that  program  in  the  investment  justification.  Therefore, 
being  aware  of  conceptual  systems  in  the  development  pipeline  that  constitute  the  capability  drivers,  and  when  they 
will  potentially  reach  MS  A,  is  essential  to  both  developing  the  proposed  plan  and  performing  the  investment 
justification. 

Once  the  vision,  target  goals,  and  timeline  have  been  defined,  the  next  step  is  to  determine  the  key  capabilities 
needed  for  successful  development  of  the  conceptual  systems  being  used  for  the  value-based  justification.  This 
involves  conducting  analyses  to  define  (1)  the  key  developmental  test  capability  areas,  (2)  the  requisite  test  objectives 
that  need  to  be  successfully  demonstrated  in  each  capability  area,  and  (3)  those  requisite  objectives  affected  by  the 
proposed  enhancements.  Identifying  the  requisite  test  objectives  includes  describing  the  parameters  that  need  to  be 
measured,  the  test  approach  and  facilities  that  would  be  employed,  the  test  conditions  and  test  variables,  the  test  articles 
required  (including  their  scale),  any  specialized  equipment  required  (such  as  specialized  instrumentation, 
measurement  devices,  or  test  fixtures)  or  specific  testing  technologies  needed  (such  as  CFD  and  other  modeling 
capabilities),  the  required  flight  conditions,  and  the  amount  of  testing  needed  (the  number  of  test  runs/test  hours/total 
time  at  conditions),  as  well  as  any  special  workforce  skills  needed  to  support  the  broad  spectrum  of  activities 
performed  during  conceptual  system  development.  Conducting  these  analyses  typically  involves  significant  and  active 
participation  by  SMEs  with  specific  knowledge  and  experience  in  system  development  and  DT&E.  The  product  of 
these  analyses  is  a  set  of  tables  (like  the  one  depicted  in  Table  1)  that  are  populated  with  information  and  data 
identifying  the  test  capability  areas  and  describing  the  test  objectives  that  should  be  satisfied  for  each  of  the  conceptual 
systems.  These  tables  are  used  to  help  identify  the  current  capability  area  gaps  that  need  to  be  closed. 
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Table  1  Key  Capability  Needs. 


T/O# 

Test  Capability  Area 

Req’d  Perl 
Parameter 

Approach/ 

Facilities 

Test 

Conditions 

Test 

Variables 

Test 

Articles 

Special 

Equip 

#Flt 

Conditions 

Test 

Hrs/Runs 

Tot 

Time 

1.0 

Aerodynamics 

1.1 

Stability  &  Control  Effects 

1.2 

Boundary  Layer  Transition 

1.3 

Inlet  Performance 

2.0 

Aerothermodynamics 

3.0 

Materials  Characterization 

4.0 

Propulsion 

5.0 

Stage  /  Stores  Separation 

6.0 

Weather  /  Erosion 

7.0 

Guidance  /  Nav  /  Control 

8.0 

System  Lethality 

9.0 

Survivability  / 

Vulnerability 

10.0 

Flight  Testing 

3.  Step  3  -  Identify  the  Capability  Gaps  -  And  the  Impacts  of  Not  Closing  Them 

After  the  DT&E  capability  requirements  for  each  of  the  conceptual  system  applications  included  in  the  justification 
analysis  have  been  determined  using  the  process  described  in  Step  2  above,  the  next  step  is  to  determine  if  capability 
gaps  exist  by  assessing  the  ability  of  the  existing  RDT&E  infrastructure  to  satisfy  the  identified  facility,  test  tools, 
testing  technologies,  and  workforce  capability  needs  in  Table  1.  Those  capability  needs  that  cannot  be  satisfied  by  the 
current  RDT&E  infrastructure  constitute  the  capability  gaps. 

The  process  of  assessing  which  capability  gaps  need  to  be  closed  begins  by  taking  each  identified  gap  and 
quantifying  the  first-order  effects  of  not  closing  that  gap  on  (1)  technical  risk,  (2)  system  design  and  development 
during  full-scale  development  (ESD)  and  DT&E,  and  (3)  system  operation  and  sustainment  (O&S)  in  the  field,  as 
shown  in  Table  2.  This  analysis  typically  involves  the  same  group  of  SMEs  that  supported  the  key  capability  needs 
analysis.  Each  SME  is  also  asked  to  provide  a  high/medium/low  assessment  as  to  the  importance  of  closing  the 
identified  capability  gaps. 


Table  2  Impact  and  Importance  of  Not  Closing  tbe  Identified  CapabiUty  Gaps. 


Gap 

No. 

Capability 

Gap 

Technical  Risk  (1*‘  Order 
Effect  on  Technology 
Development) 

System  Design/Development  Impacts 
(1“  Order  Effect  on  FSD/DT&E) 

O&S  Impacts 
(!'•  Order  Effect  on 
O&S  Capabilities) 

Importance 

C-1 

Lack  of  . . . 

Inability  to  . . . 

•  Inconsistencies 

between. . . 

•  Inability  to  predict... 

•  No  knowledge  of  . . . 

•  Reduced  confidence  in  the  ability 

to... 

•  Greater  operability  margins  would  be 
required,  thereby  increasing  weight 
and/or  reducing  performance 

•  Resolution  could  require  many  tens  of 
flight  tests  to  establish  the  required 
confidence  for  initial  fielding 

•  Reduced  system 
range  would. . . 

•  Reduced  system 
effectiveness 

would... 

High 

C-2 

Lack  of  . . . 

Inability  to  . . . 

•  Inconsistencies 

between. . . 

•  Inability  to  predict. . . 

•  No  knowledge  of  . . . 

•  Reduced  confidence  in  the  ability 

to... 

•  Greater  operability  margins  would  be 
required,  thereby  increasing  weight 
and/or  reducing  performance 

•  Resolution  could  require  many  tens  of 
flight  tests  to  establish  the  required 
confidence  for  initial  fielding 

•  Reduced  system 
range  would. . . 

•  Reduced  system 
effectiveness 

would... 

Medium 

Based  on  the  assessments  provided  by  the  SMEs  for  each  conceptual  system  application,  numeric  values  ranging 
from  1  to  10  can  be  assigned  to  the  gaps  and  a  Pareto  chart  (depicted  in  Eig.  2)  can  then  be  constructed  to  determine 
which  gaps,  if  closed,  would  have  the  greatest  overall  importance  to  each  specific  conceptual  system  application.  This 
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analysis  is  repeated  for  each  of  the  conceptual  systems  and  a  collective  set  of  capability  gaps  is  then  compiled  for  all 
the  conceptual  system  applications  included  in  the  BCA,  as  shown  in  Table  3.  (As  highlighted  in  Table  3,  a  given 
capability  gap  may  be  relevant  to  multiple  conceptual  system  applications,  but  how  the  gap  would  be  closed  may 
differ  from  application  to  application.)  This  collective  set  of  capability  gaps  forms  the  basis  for  the  proposed  plan  that 
is  evaluated  using  the  value-based  justification  process. 


GAPS 

Fig.  2  Importance  of  Not  Closing  the  Gaps  for  Conceptual  System  C. 


Table  3  Collective  Set  of  Capability  Gaps  (All  Conceptual  System  AppUcations). 


System/ 
Gap  No. 

Capability  Gap 

System/ 
Gap  No. 

Capability  Gap 

System/ 
Gap  No. 

Capability  Gap 

A-1 

Lack  of ... . 

B-1 

Lack  of ... . 

C-1 

Lack  of ... . 

A-2 

Inability  to  . . . 

B-2 

Inability  to  . . . 

C-2 

Inability  to  . . . 

A-i 

Inefficient  CFD  tools 

B-i 

Inefficient  CFD  tools 

C-i 

Inefficient  CFD  tools 

B-j 

Lack  of ... . 

C-k 

Lack  of ... . 

C.  Developing  a  Plan  for  Closing  the  Identified  Capability  Gaps 

Once  all  the  data  needed  to  prepare  the  pathway  have  been  compiled  and  analyzed,  the  next  steps  in  the  process 
are  to  (1)  develop  a  detailed  plan  that  closes  the  defined  capability  gaps  (including  an  estimate  of  its  cost),  and  (2) 
socialize  the  plan  with  the  broader  community  of  capability  owners  and  stakeholders. 

7.  Step  1  -  Develop  the  Plan 

The  first  task  in  developing  the  plan  is  to  formulate  a  set  of  projects  that  must  be  executed  to  close  the  gaps 
identified  in  Table  3.  Each  project  must  be  defined  in  sufficient  detail  (typically  one  to  three  pages  each)  to  describe 
its  goal(s),  how  it  will  be  executed  (i.e.,  the  tasks  and  sub-tasks  that  must  be  conducted  to  successfully  complete  the 
project),  its  execution  timeline,  any  interrelationship  it  has  with  other  projects  in  the  list  (i.e.,  does  it  receive  input 
from,  or  provide  input  to,  any  other  projects),  and  its  estimated  cost,  by  fiscal  year.  From  this,  a  project  listing  can  be 
constructed  of  all  the  projects  included  in  the  overall  execution  plan,  as  shown  in  Table  4. 


7 


Table  4  Project  Listing  by  Conceptual  System  Application. 


Project  Listing 

Project  No. 

Conceptual  System  A 

Project  Title  A-1: 

1 

Project  Title  A-2: 

2 

Project  Title  ...: 

Project  Title  A-X: 

X 

Conceptual  System  B 

Project  Title  B-1: 

Xh-1 

Project  Title  B-2: 

Xh-2 

Project  Title  ...: 

Project  Title  B-Y: 

Xh-Y 

Conceptual  System  C 

Project  Title  C-1: 

Xh-Yh-1 

Project  Title  C-2: 

Xh-Yh-2 

Project  Title  ...: 

Project  Title  C-Z: 

Xh-Yh-Z 

Once  the  various  projects  have  been  identified  and  defined  in  sufficient  detail,  a  cross-walk  can  then  be  performed 
between  the  projects  and  the  gaps  to  evaluate  and  assess  their  interrelationships,  as  shown  in  Table  5.  Multiple  projects 
may  support  an  individual  gap,  and  multiple  gaps  may  be  supported  by  an  individual  project  (as  highlighted  in  yellow). 
Performing  this  cross-walk  not  only  reveals  the  “robustness”  of  each  project  in  terms  of  its  relevance  to  multiple  gaps 
(as  demonstrated  by  Project  2  in  Table  5),  but  it  also  reveals  the  criticality  of  individual  projects  to  the  resolution  of  a 
specific  gap  (as  demonstrated  by  Project  7,  which  is  the  only  project  supporting  Capability  Gap  A-2).  Knowing  these 
interrelationships  can  assist  the  Program  Manager  in  assessing  and  managing  risk  during  plan  execution. 


Table  5  Projects/Capability  Gaps  Cross-Walk. 


Gap# 

Capability  Gap 

Projects 

1 

2 

3 

4 

5 

6 

7 

... 

X 

A-1 

Lack  of ... . 

X 

X 

X 

X 

X 

X 

A-2 

Inability  to  . . . 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

A-i 

Inefficient  CFD 
tools 

X 

X 

X 

X 

X 

X 

The  end-product  of  the  plan  development  effort  is  an  executable  roadmap  (as  depicted  in  Fig.  3)  showing  the 
projects  that  are  being  proposed  for  funding,  their  relevance  to  the  conceptual  systems  used  in  the  value-based 
justification,  their  time-phased  connectivity  to  the  MS  A  development  “off-ramps”  for  the  various  conceptual  systems, 
and  the  funding  both  currently  programmed  and  required  for  each  year  of  the  program  execution  schedule.  The 
difference  between  what  is  programmed  and  what  is  required  is  the  funding  augmentation  that  is  being  justified  using 
the  value-based  process. 
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Conceptual  System  Fiscal  Year 

PROGRAM  PLAN 

Pro)# 

A 

B 

c 

FY19 

FY20 

FY21 

FY22 

FY23 

FY24 

FY25 

FY26 

FY27 

FY28 

FY29 

Conceptual  System  A  Ground  T est  Enhancements 

Totals: 

($K) 

(SK) 

(SK) 

(SK) 

ProjeclTilfcA-l: 

1 

X 

($K) 

($K) 

(SK) 

(SK) 

Project  Titfe  A-2: 

2 

X 

X 

($K) 

(SK) 

(SK) 

i 

Project  Title  ...: 

X 

X 

($K) 

(SK) 

(SK) 

Project  Titfc  A-X: 

X 

X 

(SK) 

Conceptual  System  B  Ground  Test  Enhancements 

Totals: 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Title  B-1: 

X+1 

X 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Titfc  B-2: 

X+2 

X 

X 

X 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Titfe  ...: 

X 

(SK) 

(SK) 

(SK) 

Project  Title  B-Y: 

X+Y 

X 

X 

Conceptual  System  C  Ground  Test  Enhancements 

Totals: 

($K) 

(SK) 

(SK) 

(SK)  1  (SK)  1  (SK)  1  (SK)  1  (SK)  |  (SK)  |  ($K)  |  ($K) 

Project  Title  C- 1 : 

X+Y+1 

X 

X 

($K) 

(SK) 

(SK) 

Proiect  Titfe  C-2: 

X+Y+2 

X 

X 

(SK) 

(SK) 

(SK)  1  (SK)  1  (SK)  1  (SK) 

Proiect  Title  ...: 

X 

(SK)  1  (SK)  1  (SK) 

(SK) 

(SK) 

Proiect  Title  C-Z: 

X+Y+Z 

X 

X 

(SK) 

(SK) 

Flight  Test  Enhancements 

Totals: 

($K)|  (SK) 

(SK) 

(SK) 

(SK) 

(SK)  1  (SK)  1  (SK)  1  (SK) 

(SK) 

(SK) 

Project  Titl6;FT-1 

X+Y+Z+1 

X 

X 

X 

(SK) 

(SK) 

(SK) 

Project  Titl6;FT-2 

X+Y+Z+2 

X 

X 

X 

(SK) 

(SK) 

(SK) 

M&S  Enhancements 

Totals: 

($K)|  (SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Title ;M8iS-1 

X+Y+Z+3 

X 

X 

X 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Titl6;M&S-2 

X+Y+Z+4 

X 

X 

X 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Test  Tools  and  Testing  Techniques 

Totals: 

($K) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Title:  TT-1 

X+Y+Z+5 

X 

X 

X 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Title:  TT-2 

X+Y+Z+6 

X 

X 

X 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Workforce  Development 

Totals: 

($K) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Title:  WF-1 

X+Y+Z+7 

X 

X 

X 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

Project  Title:  WF-2 

X+Y+Z+8 

X 

X 

X 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

(SK) 

U.S.  ADDlications  Roadmap  (Use  Cases)  1 1 

7 

Co 

Co 

ncfiptual  Sy.stftm  A  i - C/ 

Tech  Demo 

A 

A 

POR 

J 

MSA  M  MSC 

nceotual  System  B  i - S/ 

Tech  Demo 

POR 

MSA  A 

Conceptual  System  C  I 

Tech  Demo 

POR 

MSA 

Fig.  3  Plan  Roadmap. 


2.  Step  2  -  Socialize  the  Plan 

The  first  thing  to  remember  about  the  socialization  process  is  that  an  individual  somewhere  in  the  senior  leadership 
chain  has  the  authority  to  budget  the  funding  augmentation  being  proposed — the  challenge  is  to  identify  that  person. 
That’s  the  good  news.  The  bad  news  is  that  there  are  potentially  dozens  of  mid-level  managers  along  the  path  to  that 
outcome  who  will  present  a  litany  of  reasons  why  any  effort  expended  to  justify  and  seek  a  funding  augmentation  is 
destined  to  be  fruitless — those  are  the  people  on  whom  the  socialization  effort  focuses.  They  will  say  things  like 
“budgets  are  tight  this  year”  (News-Flash  #1:  budgets  are  tight  every  year),  or  “this  is  not  on  the  list  of  top  priorities 
for  our  organization”  (News-Flash  #2:  RDT&E  infrastructure  investments  seldom  are),  or  “there  is  no  Program  of 
Record  (POR)  stating  it  requires  the  proposed  enhanced  capabilities”  (News-Flash  #3:  if  a  POR  already  exists, 
significant  infrastructure-related  capability  enhancements  are  already  late-to-need).  But  hang  in  there  and  do  not 
become  discouraged.  Nobody  said  this  was  going  to  be  easy — that  is  just  the  bureaucracy  at  work. 

The  second  thing  to  remember  is  that  the  socialization  process  is  a  marathon,  not  a  sprint.  If  the  capability 
enhancements  being  proposed  are  critically  important  to  the  nation  and/or  the  aerospace  RDT&E  community, 
visionary  leaders  will  recognize  it  and  become  supportive  advocates.  In  the  case  of  hypersonics,  those  advocates  were 
Congress  and  the  White  House! 

Since  the  initial  legwork  is  already  done  (Preparing  the  Pathway,  Steps  1  through  3),  an  advocacy  briefing  package 
should  then  be  constructed  and  meetings  scheduled  with  potential  stakeholders  and  mid-level  managers,  starting  at 
the  bottom  of  the  organization  and  working  up  the  leadership  chain.  So  this  would  be  the  time  to  begin  targeting  those 
individuals  and  getting  on  their  calendars.  The  purpose  of  these  meetings  is  to  gather  information  that  can  be  useful 
in  refining  the  proposed  investment  augmentation  justification  (both  technically  and  from  a  national  importance 
perspective).  To  that  end,  the  briefing  package  should  clearly  articulate  the  military  and/or  economic  benefits  and 
importance,  the  vision,  the  goal(s),  the  capability  needs,  and  the  plan  for  overcoming  the  identified  gaps.  During  these 
meetings,  capture  the  views  and  perspectives  offered  by  the  various  participants,  as  these  will  undoubtedly  reveal 
potential  roadblocks  to  success  (at  least  from  their  vantage  points)  that  need  to  be  addressed  as  progress  is  made  up 
the  management  chain.  Remember,  the  goal  of  socialization  is  to  generate  advocates,  so  listen  attentively  and  adjust 
accordingly!  As  a  point  of  reference,  for  the  hypersonic  effort,  socializing  the  plan  at  the  various  levels  took  over  eight 
months ! 

After  these  meeting  have  been  completed,  and  (to  the  extent  feasible)  stakeholder  and  senior  management  issues 
and  concerns  have  been  addressed,  it  is  now  time  to  quantify  the  cost  and  schedule  impacts  associated  with  not  closing 
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the  capability  gaps  that  have  been  identified  for  each  of  the  conceptual  system  development  programs  being  used  in 
the  justification  analysis.  The  reason  all  the  up-front  planning  and  socialization  efforts  are  necessary  is  because  (as 
stated  earlier)  the  value-based  justification  process  is  extremely  data-,  time-,  and  analysis-intensive,  so  “getting  all 
your  ducks  in  line”  before  starting  will  help  to  minimize  the  probability  of  having  to  do  it  a  second  time. 

D.  Quantifying  the  Programmatic  Impacts  of  Not  Closing  the  Identified  Capability  Gaps 

With  the  pathway  prepared  and  the  proposed  plan  developed  and  socialized,  the  process  of  quantifying  the 
programmatic  cost  and  schedule  impacts  caused  by  not  closing  the  identified  capability  gaps  can  commence.  This 
involves  (1)  establishing  the  set  of  baseline  assumptions  to  be  used;  (2)  identifying  representative  comparative 
programs  that  have  actual  system  development  cost  and  schedule  data  available  that  can  be  used;  (3)  gathering  the 
requisite  cost,  schedule,  and  test  frequency  data  from  the  comparative  development  programs  chosen,  and  creating  the 
requisite  cost  estimating  relationships  (CERs);  (4)  constructing  a  cost  model;  (5)  determining  the  nature  and  extent  of 
the  impact  caused  by  not  having  the  capability  enhancements;  and  (6)  using  the  cost  model  to  assess  the  programmatic 
cost  and  schedule  impacts  with  and  without  the  proposed  capability  enhancements.  These  six  steps  are  discussed  in 
greater  detail  below. 

1.  Step  1  -  Establish  the  Baseline  Analysis  Assumptions 

As  with  any  study,  establishing  the  baseline  assumptions  is  a  critical  first  step,  as  the  assumptions  strongly 
influence  the  outcomes.  In  conducting  the  hypersonic  study,  the  following  assumptions  were  made: 

(1)  The  additional  investment  for  the  proposed  capability  enhancements  (contained  in  the  plan)  is  a  given — the 
value-based  justification  process  estimates  the  return  on  that  investment; 

(2)  The  cost  of  development  programs  increases  with  the  complexity  of  the  requirements; 

(3)  Experimentation  is  critical  to  the  development  of  more  accurate  computational  capabilities; 

(4)  More  accurate  computational  capabilities  provide  greater  understanding  of  the  underlying  physics,  thus 
enabling  a  more  compressed,  lower-risk  system  development  schedule; 

(5)  Experimentation  capabilities  must  be  enhanced  to  improve  understanding  of  the  underlying  physics; 

(6)  failure  to  provide  the  requisite  Tc&E  facility  and  capability  enhancements  will  increase  the  number  of 
undetected  design  flaws  that  occur  during  system  development; 

(7)  All  proposed  enhancements  will  be  available  at  MS  A  for  the  conceptual  programs  being  evaluated; 

(8)  Relevant  past  programs  (used  to  construct  the  cost  model)  had  an  adequate  Tc&E  infrastructure  available,  so 
actual  experience  from  these  programs  can  be  used  to  inform  the  baseline  conceptual  system  development 
programs  used  (and  executed  with  the  equivalent  of  an  enhanced  T&E  infrastructure)  in  the  analysis; 

(9)  Actual  experience  from  relevant  past  programs  can  be  used  to  inform  certain  programmatic  cost  elements 
for  future  programs,  such  as  the 

a.  Number  of  flight  tests, 

b.  Time  between  flight  tests  (test  phases), 

c.  Development  time  from  MS  A  through  MS  C, 

d.  Schedule  growth,  and 

e.  Costs  for  the  prime  development  contract  (including  recurring  and  non-recurring  costs,  DT&E, 
and  flight  testing);  and 

(10)  Design  flaws  encountered  by  using  unenhanced  experimentation  and  computational  capabilities  will  cause 
program  delays,  adding  cost  at  the  time  they  are  discovered. 

The  assumptions  above  would  be  relevant  to  most  value-based  justification  analyses;  nonetheless,  any  study 
assumptions  must  be  tailored  to  each  new  capability  enhancement  being  proposed  for  a  BCA. 

2.  Step  2  —  Identify  Representative  Comparative  Programs 

Identifying  representative  comparative  development  programs  starts  with  determining  which  programs  most 
closely  align  with  the  operational  capabilities  afforded  by  the  conceptual  systems  being  used  in  the  analysis.  Eor 
example,  representative  programs  could  include  development  of  the  Boeing  767  and  777  in  the  civil  arena,  or 
development  of  the  E-22  and  Joint  Air-to-Surface  Standoff  Missile  (JASSM)  in  the  military  arena.  Some  of  the  criteria 
that  need  to  be  critically  assessed  for  the  comparative  development  programs  include  (1)  whether  they  were 
successfully  developed  and  fielded,  (2)  whether  they  have  similar  operational  capabilities  or  capability  attributes 
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(including  similar  technology  reach)  to  the  conceptual  development  systems  being  used,  and  (3)  whether  they  have 
similarly  reported  cost  and  schedule  data.  It  is  worth  emphasizing  that  this  third  criterion  can  pose  several  problems 
for  your  study  team.  If  actual  cost  data  are  not  available,  an  accurate  cost  estimate  cannot  be  developed.  Further,  the 
assistance  and  counsel  of  SMEs  with  access  to  such  information  and  data  is  essential.  This  is  vital  to  the  successful 
conduct  of  a  value-based  justification  analysis. 

3.  Step  3  -  Gather  the  Requisite  Data  and  Create  the  Requisite  CERs 

To  evaluate  the  programmatic  cost  and  schedule  impacts  on  the  conceptual  systems  being  used  in  the  analysis 
(Step  6),  the  cost  and  schedule  driver(s)  first  need  to  be  identified.  In  the  high-speed  missile  analysis,  the  driver  was 
undetected  flaws  encountered  during  flight  test,  which  would  result  in  program  delays  and  added  cost  at  the  time 
encountered. 

Cost  and  schedule  data  for  the  representative  comparative  development  programs  must  span  the  spectrum  of 
potential  system  applications  that  would  be  affected  by  the  new  capabilities  in  the  proposed  plan.  Schedule  data  must 
include  the  initially  proposed  schedule  that  preceded  the  start  of  system  development  as  well  as  the  actual  final 
schedule  after  successful  completion  of  system  development.  These  schedule  data  are  used  to  evaluate  the 
reasonableness  of  any  projected  program  slippages  calculated  by  the  cost  model  used  in  the  evaluation  of  the  MS  A 
through  MS  C  conceptual  system  development  programs. 

In  addition  to  the  recurring  unit  cost  for  each  representative  system,  cost  allocation  data  should  include  both  the 
recurring  and  non-recurring  development  costs,  other  contract  costs,  other  government  costs  (for  government-funded 
programs),  developmental  test  costs,  and  the  cost  of  ground  and  flight  testing.  We  show  examples  of  the  requisite 
development  cost  allocation  and  schedule  CERs  needed  to  construct  the  cost  model  in  Eig.  4  and  Eig.  5,  respectively. 


m 

w 

3 

> 


■  T  otal 
Development 

■  Contract 

■  Total  Munition 

■  Non  Recurring 

■  Recurring 

■  Ground  Test 
Flight  Test 


'2M  ~4M 


~14M 


Munition  Recurring  Unit  Cost  in  $M  (notto  scale) 


Fig.  4  Allocation  of  Development  Costs. 
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In  addition  to  these  CERs,  actual  data  are  needed  on  the  number  of  flight  tests  that  were  conducted  between  MS 
A  and  MS  C  for  the  representative  successful  system  development  programs  selected  (including  any  preceding 
unsuccessful  programs  that  contributed  materially  to  the  system  design  or  were  instrumental  in  the  program’s  ultimate 
success)  as  well  as  the  frequency  of  flight  tests  across  the  spectrum  of  all  representative  comparative  programs  used 
in  the  analysis.  These  data  are  used  to  not  only  predict  the  number  of  flight  tests  conducted  in  the  cost  model,  but  also 
to  estimate  the  characteristic  times  between  flight  test  centers,  so  estimates  of  projected  program  delays  can  be 
determined  using  actual  historic  flight  test  data.  Examples  of  the  estimating  relationship  for  the  number  of  flight  tests 
required  and  the  flight  test  frequency  histogram  are  shown  in  Eig.  6  and  Eig.  7,  respectively. 


Fig.  6  Number  of  Flight  Tests  (MS  A  through  MS  C). 
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4.  Step  4  -  Construct  the  Cost  Model 

Construction  of  the  cost  model  involves  incorporating  all  the  information  and  data  gathered  during  Steps  2  and  3 
(above)  into  a  probabilistic  model  that  can  be  used  to  generate  a  cost-schedule  profile  based  on  the  collective 
representative  comparative  program  data.  Fig.  8  shows  an  example  cost-schedule  profile. 

Because  constructing  and  exercising  models  of  this  type  requires  substantial  expertise  and  experience  in  both  the 
formulation  and  execution  of  system  development  programs,  as  well  as  in  cost  analysis  data  and  methods,  seeking  the 
counsel  and  assistance  of  SMEs  knowledgeable  in  this  field  is  highly  recommended.  Once  the  model  has  been 
developed  and  validated,  one  can  then  tailor  it  to  each  conceptual  development  program  used  in  the  value-based 
justification  analysis  to  determine  the  cost-schedule  impacts  of  unanticipated  design  flaws  resulting  from  not  closing 
the  identified  capability  gaps  being  addressed  in  the  plan.  First,  however,  some  additional  counsel  is  needed  from  the 
SMEs  who  helped  to  construct  Table  1,  Table  2,  and  Table  5  with  regard  to  the  nature  and  extent  of  the  impact  caused 
by  not  having  the  requisite  T&E  infrastructure. 


■  FlightTestCost 

■  GroundTestCost 

■  GovernmentDTCost 

■  OtherGovernmentCosts 

■  OtherContractCosts 

■  NRDevelopmentCost 

■  RecurringDevelopmentCost 


Fig.  8  Example  Cost/Schedule  Profile. 


5.  Step  5  —  Determine  the  Nature  and  Extent  of  the  Impact  Caused  by  Not  Having  the  Capability  Enhancements 

As  noted  earlier,  in  the  high-speed  missile  arena,  the  nature  of  the  impact  that  drove  program  cost  and  schedule 
was  the  occurrence  of  undetected  flaws  during  flight  test.  Determining  the  extent  of  the  impact  (i.e.,  the  number  of 
undetected  flaws  encountered  during  flight  test)  due  to  not  having  available  the  requisite  RDT&E  capabilities  (once 
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again)  required  input  from  SMEs  with  specific  knowledge  and  experience  in  system  development  and  DT&E.  These 
SMEs  provided  informed  determinations,  based  on  their  best  engineering  judgment,  regarding  the  ability  of  currently 
available  capabilities  to  satisfy  the  developmental  test  objectives  identified  in  Table  1.  Eor  each  test  objective  (or 
group  of  test  objectives),  the  SMEs  provided  informed  judgments  on  which  test  objectives  would  be  affected,  and  in 
what  ways  (test  times/test  costs/  frequency  of  test  campaigns),  by  not  having  available  the  proposed  capability 
enhancements.  They  also  provided  informed  judgments  on  which  capability  shortfalls  would  potentially  result  in  the 
existence — and  number — of  undetected  flaws.  We  present  an  example  of  this  analysis  in  Table  6. 


Table  6  Example  Analysis  of  Estimated  Undetected  Design  Flaws. 


Conceptual  System  A  (with  Enhancements) 

Test  Type 

Test 

Objectives 

Addressed 

Est  Test 
Cost 
($K) 

Est  Test 
Time 
(weeks) 

Number  of  Ground  Tests 

Total 

Cost 

($K) 

Experimental 
(Supplements  Data) 

Undetected  Design 
Flaws  (Possible  F/T 
Failures) 

Pre- 
MS  A 

MS  A-B 

Post  MS  B 

MS  A-B 

MS  B-C 

MS  A-B 

MS  B-C 

Aero 

l.l-to-1.5 

4,000 

8 

2 

2 

0 

16,000 

baseline 

baseline 

baseline 

baseline 

Aerotherm 

2.1-to-2.7 

1,000 

4 

1 

1 

0 

2,000 

baseline 

baseline 

baseline 

baseline 

Materials 

3.4-to-3.11 

2,000 

26 

2 

1 

0 

6,000 

baseline 

baseline 

baseline 

baseline 

Propulsion 

4.2-to-4.3 

5,000 

12 

2 

2 

0 

20,000 

baseline 

baseline 

baseline 

baseline 

Stage/Store 

5.1 

500 

2 

0 

2 

8 

5,000 

baseline 

baseline 

baseline 

baseline 

Weather 

6.1-to-6.3 

2,500 

12 

0 

2 

2 

10,000 

baseline 

baseline 

baseline 

baseline 

GNC 

7.5-to-7.7 

2,000 

8 

0 

2 

2 

8,000 

baseline 

baseline 

baseline 

baseline 

Lethality 

8.1 

1,000 

8 

0 

1 

2 

3,000 

baseline 

baseline 

baseline 

baseline 

Conceptual  System  A  (without  Enhancements) 

Test  Type 

Test 

Objectives 

Addressed 

Est  Test 
Cost 
($K) 

Est  Test 
Time 
(weeks) 

Number  of  Ground  Tests 

Total 

Cost 

($K) 

Experimental 
(Supplements  Data) 

Undetected  Design 
Flaws  (Possible  F/T 
Failures) 

Pre- 
MS  A 

MS  A-B 

Post  MS  B 

MS  A-B 

MS  B-C 

MS  A-B 

MS  B-C 

Aero 

l.l-to-1.5 

5,000 

10 

3 

2 

1 

30,000 

1 

1 

Aerotherm 

2.1-to-2.7 

2,000 

8 

2 

1 

0 

6,000 

Materials 

3.4-to-3.11 

2,500 

34 

2 

1 

0 

7,500 

1 

Propulsion 

4.2-to-4.3 

7,000 

18 

2 

2 

1 

35,000 

1 

Stage/Store 

5.1 

500 

2 

0 

2 

12 

7,000 

1 

Weather 

6.1-to-6.3 

2,500 

12 

0 

3 

3 

15,000 

2 

4 

GNC 

7.5-to-7.7 

2,000 

8 

0 

2 

3 

10,000 

Lethality 

8.1 

1,000 

8 

0 

1 

3 

4,000 

Identifying  and  quantifying  the  nature  and  extent  of  the  impacts  caused  by  not  having  the  requisite  RDTc&E 
capabilities  available  varies  from  application  to  application,  so  Step  5  will  have  to  be  tailored  to  the  specific  capability 
enhancements  being  proposed  and  assessed. 

6.  Step  6  -  Assess  the  Programmatic  Impacts  with  and  without  the  Proposed  Capability  Enhancements 

The  IDA  study  team  tailored  the  cost  model  developed  in  Step  4  to  create  probabilistic  models  for  each  conceptual 
system  by  incorporating  system-specific  estimates  (based  on  engineering  judgment)  of  the  time  interval  between  flight 
test  centers  and  the  length  of  the  delays  that  would  result  from  encountering  the  undetected  flaws  projected  in 
Table  6.  In  analyzing  the  potential  cost  and  schedule  impacts  in  the  high-speed  missile  analysis,  the  IDA  study  team 
assumed  that  past  successful  development  programs  had  an  adequate  infrastructure  available  to  them  from  MS  A 
through  MS  C  of  system  development.  Therefore,  the  baseline  tailored  model  for  each  conceptual  system  was  called 
the  “enhanced”  model. 

The  IDA  team  then  probabilistically  introduced  the  estimated  number  of  undetected  flaws  (and  the  projected  delays 
the  flaws  would  cause)  into  the  enhanced  model  and  ran  the  model  again  to  evaluate  the  “unenhanced”  cost-schedule 
profile.  The  team  made  multiple  computer  runs  (numbering  in  the  hundreds)  to  generate  average  outcomes  for  both 
the  enhanced  and  unenhanced  cases.  The  difference  in  cost  and  schedule  between  the  enhanced  and  unenhanced  model 
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runs,  depicted  in  Fig.  9,  represents  the  projected  savings  that  would  accrue  from  having  the  proposed  capability 
enhancements  available  for  each  conceptual  system  at  the  beginning  of  MS  A.  (In  the  Fig.  9  conceptual  example,  the 
time  savings  were  three  years  and  the  cost  savings  were  $235  million.)  The  flight  test  events  for  the  enhanced  and 
unenhanced  cases  are  depicted  as  triangular  “hats”  along  the  top  of  each  cost-schedule  profile.  The  composite  total 
cost  savings  for  all  the  conceptual  systems  used  in  the  analysis,  minus  the  estimated  up-front  cost  of  the  plan,  represent 
the  total  net  savings  that  would  accrue  from  making  the  capability  enhancement  investments. 


Enhanced 


■  RecurringOevelopmentCost  ■  NR[>evelopmentCost  ■OtherContractCosts 


■  OtherGovemmentCosts  BGovernmcntOTCost  BGroundTestCost 

■  Fi«htTestCost 


Months 


Unenhanced 


■  RecurringOevelopmentCost  iNROevelopmentCost  ■  OtherContractCosts 

■  OtherGovernmentCosts  ■  GovemmentDTCost  ■  GroundTestCost 

•  FlightTestCost 


Months 


Fig.  9  Development  Cost/Schedule  Growth  Resulting  from  Not  Closing  the  Capability  Gaps. 


The  example  analysis  described  above  was  tailored  to  a  high-speed  missile  application.  Since  each  application  will 
be  somewhat  different,  the  cost  drivers  associated  with  the  specific  conceptual  systems  being  evaluated  will  have  to 
be  determined  to  ensure  that  the  model  constructed  has  the  capability  to  respond  to  changes  in  those  drivers. 

III.  Summary 

The  purpose  of  this  paper  is  to  share  this  successful  process  for  justifying  proposed  capability  investments  with 
the  broader  aerospace  RDT&E  community,  so  that  it  can  be  employed  in  other  RDT&E  capability  areas  to  support 
both  sustainment  and  capital  investment  decisions.  Potential  users  of  this  process,  however,  need  to  realize  that  the 
process  takes  substantial  time  and  effort — and  that  success  is  not  guaranteed.  In  the  high-speed  missile  case  study,  for 
example,  the  process  described  in  Sections  II.  A  through  II. C  took  over  three  years  to  complete  (about  one-quarter  of 
which  was  associated  with  the  socialization  process),  and  it  required  substantial  effort  not  only  by  the  core  IDA  study 
team,  but  also  by  an  extensive  support  team  of  government  and  industry  SMEs  who  provided  information  and  counsel 
on  the  key  capability  needs,  the  capability  gaps,  the  impacts  of  not  closing  the  gaps,  and  the  proposed  investment  plan. 


15 


In  addition,  the  process  described  in  Section  II.D  took  another  nine  months  to  complete  and  required  substantial 
archival  research  on  the  cost,  schedule,  and  test  frequency  of  comparative  development  programs.  It  then  took  an 
additional  year  to  develop  the  Issue  Paper  seeking  the  budget  augmentation,  coordinate  it  up  to  the  Under  Secretary 
of  Defense  level  for  inclusion  in  the  President’s  Budget  Request,  shepherd  the  requested  augmentation  through  the 
various  congressional  authorization  and  appropriation  committees,  and  finally  get  the  funds  appropriated.  Thus,  the 
overall  time-period  from  the  beginning  of  the  initial  study  effort  to  get  the  attention  of  the  decision-makers,  until  the 
first  dollar  of  new  funding  was  received,  was  five  calendar  years.  It  suffices  to  say  that  the  application  of  this  process 
to  other  capability  areas  will  require  the  investment  of  substantial  time,  cost,  commitment,  and  perseverance — ^but  the 
fruits  of  a  successful  justification  can  be  significant! 
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14.  ABSTRACT 

Section  1071  of  the  National  Defense  Authorization  Act  for  Fiscal  Year  2013  required  the  Director  of  the  White  House  Office  of  Science  and  Technology  Policy  (OSTP), 
working  with  the  Secretary  of  Defense  and  the  Administrator  of  the  National  Aeronautics  and  Space  Administration  (NASA),  to  conduct  a  study  on  the  ability  of  the 
national  test  and  evaluation  (T&E)  infrastructure  to  effectively  and  efficiently  mature  hypersonic  technologies  for  defense-related  systems  development  in  the  short  and 
long  term.  It  further  required  the  Secretary  of  Defense  to  submit  a  report  to  the  appropriate  congressional  committees  containing  the  results  of  this  study,  together  with  a 
plan  for  requirements  and  proposed  investments  to  meet  Department  of  Defense  (DoD)  needs  through  2030.  The  Institute  for  Defense  Analyses  (IDA)  supported  both  the 
OSTP  and  DoD  efforts,  and  was  subsequently  tasked  to  quantify  the  projected  savings  that  could  accrue  to  each  of  three  conceptual  conventional  hypersonic  system 
development  programs  if  the  proposed  T&E  capability  enhancements  presented  in  the  DoD  plan  submitted  to  Congress  were  available  at  the  beginning  of  Milestone 
(MS)  A.  As  part  of  its  analysis,  IDA  utilized  an  IDA-developed  approach  for  valuing  the  closure  of  critical  T&E  capability  gaps  based  on  the  potential  programmatic  cost 
and  schedule  savings  that  could  reasonably  be  expected  to  accrue  from  the  start  of  MS  A  through  the  end  of  MS  C  by  funding  the  5-year,  $350M  T&E  infrastructure 
investment  proposed  in  the  DoD  plan.  This  expansion  of  the  cost-benefit  “control  volume”  to  include  projected  system  development  savings  proved  successful  in 
justifying  and  securing  full  funding  for  the  proposed  DoD  plan.  This  paper  summarizes  the  IDA  approach  and  its  application  to  making  the  business  case  for  closing 
capability  gaps  in  other  aerospace  research,  development,  test  and  evaluation  (RDT&E)  areas. 
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